Virtually visualizing energy

ABSTRACT

The techniques describe herein use sensor(s) to scan a real-world environment and obtain data associated with geometry of the real-world environment that affects how energy propagates (e.g., locations of spatial objects in a room). The sensor(s) also detect energy (e.g., sound) in the real-world environment, from which a location of a source of the energy can be determined. The techniques combine the geometry data and the energy data to determine how the detected energy propagates from the location of the source through the real-world environment. The techniques can then cause a representation of the propagating energy to be displayed, to a user, as virtual content via a mixed reality device. Accordingly, a user is able to see energy that is otherwise invisible.

BACKGROUND

Virtual reality is a technology that leverages computing devices togenerate environments that simulate physical presence in physical,real-world scenes or imagined worlds (e.g., virtual scenes) via adisplay of a computing device. In virtual reality environments, socialinteraction can be achieved between computer-generated graphicalrepresentations of a user or the user's character (e.g., an avatar) in acomputer-generated environment. Mixed reality is a technology thatmerges real and virtual worlds. Mixed reality is a technology thatproduces mixed reality environments where a physical, real-world personand/or objects in physical, real-world scenes can co-exist with avirtual, computer-generated person and/or objects in real time. Forexample, a mixed reality environment can augment a physical, real-worldscene and/or a physical, real-world person with computer-generatedgraphics (e.g., a dog, a castle, etc.) in the physical, real-worldscene.

SUMMARY

This disclosure describes techniques for enabling the use of sensor(s)to scan a real-world environment and obtain data associated withgeometry of the real-world environment that affects how energypropagates (e.g., locations of spatial objects in a room). The sensor(s)can also detect energy (e.g., sound) in the real-world environment, fromwhich a location of a source of the energy can be determined. Thetechniques combine the geometry data and the energy data to determinehow the detected energy propagates from the location of the sourcethrough the real-world environment. The techniques can then cause arepresentation of the propagating energy to be displayed, to a user, asvirtual content via a mixed reality device. Accordingly, a user is ableto see energy that is otherwise invisible.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key or essentialfeatures of the claimed subject matter, nor is it intended to be used tolimit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The Detailed Description is set forth with reference to the accompanyingfigures, in which the left-most digit of a reference number identifiesthe figure in which the reference number first appears. The use of thesame reference numbers in the same or different figures indicatessimilar or identical items or features.

FIG. 1 is a schematic diagram showing an example environment in whichinvisible energy can be virtualized and viewed by a user in a mixedreality scene.

FIG. 2 is a schematic diagram showing an example of a head mounted mixedreality display device.

FIG. 3 is a schematic diagram showing an example of a first person viewof virtual energy in a mixed reality scene.

FIG. 4 is a schematic diagram showing an example of a grid representingthe propagation of energy through a three-dimensional space.

FIG. 5 is a flow diagram that illustrates an example process todetermine a representation of how sensed energy propagates from alocation of a source through the physical, real-world environment and tovisualize the representation.

FIG. 6 is a flow diagram that illustrates an example process tore-compute the representation of how the sensed energy propagates fromthe location of the source of the energy through the environment basedon a viewing perspective of a user.

DETAILED DESCRIPTION

The techniques describe herein use sensor(s) to scan a real-worldenvironment and obtain data associated with geometry of the real-worldenvironment that affects how energy propagates (e.g., locations ofspatial objects in a room). The sensor(s) also detect energy (e.g.,sound) in the real-world environment, from which a location of a sourcecan be determined. The techniques combine the geometry data and theenergy data to determine how the detected energy propagates from thelocation of the source through the real-world environment. Thetechniques can then cause a representation of the propagating energy tobe displayed, to a user, as virtual content via a mixed reality device.

For the purposes of this discussion, physical, real-world objects (“realobjects”) or physical, real-world people (“real people” and/or “realperson”) describe objects or people, respectively, that physically existin a physical, real-world environment or scene (“real scene”) associatedwith a mixed reality display. Real objects and/or real people can movein and out of a field of view based on movement patterns of the realobjects and/or movement of a user and/or user device. Virtual,computer-generated content (“virtual content”) can describe content thatis generated by one or more computing devices to supplement the realscene in a user's field of view. In at least one example, virtualcontent can include one or more pixels each having a respective color orbrightness, and the pixels are collectively presented on a display suchto represent a person or an object that is not physically present in areal scene.

Moreover, as further discussed herein, the pixels can also representenergy that is physically present in a real scene but that typicallycannot be seen by a user. For example, the energy being represented caninclude sound, as further described in some examples below. In addition,the techniques described herein can also be used to display dynamicvolumetric visualization of Wi-Fi signals, heat, light, fluid dynamics,electro-magnetic fields, radiation, air currents, and other forms ofenergy that propagates through a three-dimensional space. In variousimplementations, a source of energy can be a real source (e.g., musicplaying, an actual burning fire, etc.). In alternative implementations,a source of energy can be a virtual source (e.g., the sound of a virtualballoon popping, heat from a virtual fire, etc.).

Illustrative Environments

FIG. 1 is a schematic diagram showing an example environment 100 inwhich a user can see invisible energy within a mixed realityenvironment, or a mixed reality scene that includes real content (e.g.,actual objects in a room) and virtual content (e.g., invisible energy).That is, a device associated with the user is configured to receivevisualization data representative of the invisible energy and displaythe visualization data for the user to view. The visualization data canbe displayed as virtual content to create, or contribute to, the mixedreality environment. For example, the visualization data can bepresented via a display of the device as the user looks through thedisplay (e.g., a transparent display) to view a physical, real-worldscene.

The example environment 100 can include a service provider 102, one ormore networks 104, one or more users 106 (e.g., user 106A, user 106B,user 106C) and one or more devices 108 (e.g., device 108A, device 108B,device 108C) associated with the one or more users 106. The serviceprovider 102 can comprise any entity, server, platform, console,computer, etc., that enables individual users (e.g., user 106A, user106B, user 106C) to visualize invisible energy in a mixed realityenvironment. The service provider 102 can be implemented in anon-distributed computing environment or can be implemented in adistributed computing environment, possibly by running some modules ondevices 108 or other remotely located devices. As shown, the serviceprovider 102 can include one or more server(s) 110, which can includeone or more processing unit(s) (e.g., processor(s) 112) andcomputer-readable media 114, such as memory. In various examples, theservice provider 102 can receive data from one or more sensors. The datareceived from the sensors can be associated with (i) geometry of aphysical, real-world environment (e.g., locations of spatial objects ina room) that affects how energy propagates and/or (ii) the energy (e.g.,a location of the source within the room, an intensity or frequency ofthe energy, etc.). Based at least in part on receiving the data, theservice provider 102 can determine how the sensed energy propagates fromthe location of the source through the physical, real-world environmentbased at least in part on the geometry, and then the service provider102 can transmit data representative of the propagating energy to thevarious devices 108 so the data can be displayed, to a user, as virtualcontent in a mixed reality environment.

In some examples, the networks 104 can be any type of network known inthe art, such as the Internet. Moreover, the devices 108 cancommunicatively couple to the networks 104 in any manner, such as by aglobal or local wired or wireless connection (e.g., local area network(LAN), intranet, etc.) and/or short range communications (e.g.,Bluetooth, etc.). The networks 104 can facilitate communication betweenthe server(s) 110 and the devices 108 associated with the users 106.

Examples of device(s) that can be included in the one or more server(s)110 can include one or more computing devices that operate in a clusteror other clustered configuration to share resources, balance load,increase performance, provide fail-over support or redundancy, or forother purposes. Device(s) included in the one or more server(s) 110 canrepresent, but are not limited to, desktop computers, server computers,web-server computers, personal computers, mobile computers, laptopcomputers, tablet computers, wearable computers, implanted computingdevices, telecommunication devices, automotive computers, networkenabled televisions, thin clients, terminals, game consoles, gamingdevices, work stations, media players, digital video recorders (DVRs),set-top boxes, cameras, integrated components for inclusion in acomputing device, appliances, or any other sort of computing devices.

Device(s) that can be included in the one or more server(s) 110 caninclude any type of computing device having one or more processingunit(s) (e.g., processor(s) 112) operably connected to computer-readablemedia 114 such as via a bus, which in some instances can include one ormore of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCIbus, and any variety of local, peripheral, and/or independent buses.Executable instructions stored on computer-readable media 114 caninclude, for example, an input module 116, a geometry module 118, anenergy module 120, a presentation module 122, and one or moreapplications 124, as well as other modules, programs, or applicationsthat are loadable and executable by the processor(s) 112.

Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-Programmable Gate Arrays(FPGAs), Application-Specific Integrated Circuits (ASICs),Application-Specific Standard Products (ASSPs), System-On-a-Chip systems(SOCs), Complex Programmable Logic Devices (CPLDs), etc. Device(s) thatcan be included in the one or more server(s) 110 can further include oneor more input/output (I/O) interface(s) coupled to the bus to allowdevice(s) to communicate with other devices such as input peripheraldevices (e.g., a keyboard, a mouse, a pen, a game controller, a voiceinput device, a touch input device, a gestural input device, a trackingdevice, a mapping device, an image camera, a depth sensor, aphysiological sensor, a microphone or other acoustic sensor, athermometer, a Geiger counter, and the like) and/or output peripheraldevices (e.g., a display, a printer, audio speakers, a haptic output,and the like). Such network interface(s) can include one or more networkinterface controllers (NICs) or other types of transceiver devices tosend and receive communications over a network.

Processing unit(s) (e.g., processor(s) 112) can represent, for example,a CPU-type processing unit, a GPU-type processing unit, an HPU-typeprocessing unit, a Field-Programmable Gate Array (FPGA), another classof digital signal processor (DSP), or other hardware logic componentsthat can, in some instances, be driven by a CPU. In various examples,the processing unit(s) (e.g., processor(s) 112) can execute one or moremodules and/or processes to cause the server(s) 110 to perform a varietyof functions, as set forth above and explained in further detail in thefollowing disclosure. Additionally, each of the processing unit(s)(e.g., processor(s) 112) can possess its own local memory, which alsocan store program modules, program data, and/or one or more operatingsystems.

In at least one configuration, the computer-readable media 114 of theserver(s) 110 can include components that facilitate interaction betweenthe service provider 102 and the one or more devices 108. The componentscan represent pieces of code executing on a computing device. In atleast some examples, the modules can be executed as computer-readableinstructions, various data structures, and so forth via at least oneprocessing unit(s) (e.g., processor(s) 112) so the service provider 102can determine how the sensed energy propagates from the location of thesource through the physical, real-world environment based at least inpart on the geometry (e.g., the sensed spatial objects in a room).Functionality to perform these operations can be included in multipledevices or a single device.

Depending on the configuration and type of the server(s) 110, thecomputer-readable media 114 can include computer storage media and/orcommunication media. Computer storage media can include volatile memory,non-volatile memory, and/or other persistent and/or auxiliary computerstorage media, removable and non-removable computer storage mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules, orother data. Computer memory is an example of computer storage media.Thus, computer storage media includes tangible and/or physical forms ofmedia included in a device and/or hardware component that is part of adevice or external to a device, including but not limited torandom-access memory (RAM), static random-access memory (SRAM), dynamicrandom-access memory (DRAM), phase-change memory, read-only memory(ROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), flash memory, compactdisc read-only memory (CD-ROM), digital versatile disks (DVDs), opticalcards or other optical storage media, miniature hard drives, memorycards, magnetic cassettes, magnetic tape, magnetic disk storage,magnetic cards or other magnetic storage devices or media, solid-statememory devices, storage arrays, network attached storage, storage areanetworks, hosted computer storage or any other storage memory, storagedevice, and/or storage medium that can be used to store and maintaininformation for access by a computing device.

In contrast, communication media can embody computer readableinstructions, data structures, program modules, or other data in amodulated data signal, such as a carrier wave, or other transmissionmechanism. The term “modulated data signal” means a signal that has oneor more of its characteristics set or changed in such a manner as toencode information in the signal. Such signals or carrier waves, etc.can be propagated on wired media such as a wired network or direct-wiredconnection, and/or wireless media such as acoustic, RF, infrared andother wireless media. As defined herein, computer storage media does notinclude communication media. That is, computer storage media does notinclude communications media consisting solely of a modulated datasignal, a carrier wave, or a propagated signal, per se.

The input module 116 is configured to receive data from one or moreinput peripheral devices (e.g., a keyboard, a mouse, a pen, a gamecontroller, a voice input device, a touch input device, a gestural inputdevice, a tracking device, a mapping device, an image camera, a depthsensor, a physiological sensor, a microphone or other acoustic sensor, athermometer, a Geiger counter, a color sensor, a gravity sensor and thelike). In some examples, the one or more input peripheral devices can beintegrated into the one or more server(s) 110 and/or other machinesand/or devices 108. In other examples, the one or more input peripheraldevices can be communicatively coupled to the one or more server(s) 110and/or other machines and/or devices 108. For instance, the inputperipheral devices can be distributed throughout a physical, real-worldenvironment (e.g., placed at particular locations within a room in whicha mixed reality scene is to be implemented). The one or more inputperipheral devices can be associated with a single device (e.g.,MICROSOFT® KINECT®, INTEL® Perceptual Computing SDK 2013, LEAP MOTION®,etc.) or separate devices.

The input module 116 is configured to receive data associated withgeometry of a physical, real-world environment (e.g., athree-dimensional representation of a room). For example, the geometrycan include mapped spatial real objects actually located in thephysical, real-world environment (e.g., furniture, people, etc.) and/ordefined boundaries of the physical, real-world environment (e.g., thewalls of a room, the floor of the room, the ceiling of a room, etc.)outside of which objects may no longer be sensed. In various examples,some of the geometry can be stationary (e.g., fixed with little or nomovement). Thus, the received data associated with the geometry of aphysical, real-world environment can be obtained based on an initialenvironment scan, where the server(s) 110 instruct the peripheral inputdevices (e.g., sensors) to obtain data regarding the geometry of thephysical, real-world environment. However, in other examples, at leastsome of the geometry can be dynamic (e.g., a spatial object may movewithin a room) and the data can be obtained based on instructions toperform scheduled or periodic scans over a period of time (e.g., everyminute, every ten minutes, every hour, every day, etc.) to account forany changes and to update the geometry. In one example, a scan can beperformed in real-time based on, or in response to, detected energy in aphysical, real-world environment.

In various examples, based on the received data associated with geometryof a physical, real-world environment, the geometry module 118 isconfigured to compute energy-based geometry parameter fields. Forinstance, the geometry module 118 can compute parameter fields whichencode scene level properties such as one or more of a loudness map(e.g., for sound propagation), a reflection map to model how energy isredirected in response to encountering a real or a virtual object, areverberation map to model how repeated energy (e.g., an echo)propagates, an amount of time it takes energy to travel from a sourcelocation (e.g., a first three-dimensional point) in the real scene toanother three-dimensional point within the real scene, etc. Theenergy-based geometry parameter fields can be used to determine howenergy, detected at a particular location, propagates through thephysical, real-world environment. For instance, for a sound sourcelocation, the geometry module 118 can compute a parameter field mapwhich encodes how sound propagates in the three-dimensional space of areal scene, and thus, information regarding how loud the sound would beat different points in that space can be extracted based on the soundsource location.

The input module 116 is further configured to receive data associatedwith energy (e.g., virtual or real) that is present in, or has beennewly introduced to, the physical, real-world environment. The dataassociated with the energy can be indicative of: a location of a sourcethat produces the energy (e.g., via the use of angles of detection), anintensity of the energy, and/or a frequency of the energy. In oneexample, an array of sensors can be used to localize energy (e.g.,sound) by triangulating a source. That is, individual sensors orindividual sensing devices distributed throughout the physical,real-world environment can report, to the input module 116, one or moreof: an angle at which an energy source is detected, an intensity of theenergy detected (e.g., an amplitude of sound), a time instance at whicha sensor detects a signal, and/or a frequency spectrum of the energydetected (e.g., a frequency of the detected sound). In some instances,sensors or sensing devices can be co-located to triangulate an energysource. In one example, to perform triangulation on sound, at leastthree separate sensors or sensing devices with a microphone can be usedto detect the sound so the sound can be localized. In at least oneexample, the triangulation can be solved by a least squares minimizationapproach, but other alternative approaches can also be used.

The energy module 120 is configured to combine the received dataassociated with the detected energy with the energy-based geometryparameter fields to determine how the energy propagates from the sourcelocation through the physical, real-world environment (e.g., how theenergy interacts with the geometry of the physical, real-worldenvironment). In some examples, the energy module 120 can also determinehow the energy interacts with virtual objects in the physical,real-world environment, as well.

The presentation module 122 is configured to send rendering data todevices 108 for presenting virtual content (e.g., the detected energyand how it propagates through a mixed reality scene) via the devices108. Accordingly, the presentation module 122 is configured to visualizethe energy and provide the visualized energy to the devices 108 so thedevices 108 can render and/or display the energy and the users 106 canview invisible energy in a mixed reality scene. For example, thepresentation module 122 can visualize the energy so that a viewer cansee the intensity of the energy, the frequency of the energy, theeffects of the geometry on the energy and/or the time delay effects ofthe energy. In one example, the presentation module 122 uses athree-dimensional structure to represent the energy as a set (e.g., athree-dimensional array) of spheres distributed over the space of thephysical, real-world environment (as shown in the “grid” of FIG. 4). Forinstance, perceptual parameters can be encoded based on the following:the size (e.g., radius) of a sphere indicates intensity (e.g., loudness)of the energy (e.g., spheres with a larger radius represent sound thatis louder than the sound represented by spheres with a smaller radius),the color of a sphere indicates frequency of the energy (e.g., variousfrequencies or frequency ranges can be assigned different colors—red,green and blue) and the radius of the spheres can be modulated to showthe effect of the geometry (e.g. the radius of the spheres decays as thedistance from a particular sphere to the source increases).

In some instances, the energy can be visualized based on a perspectiveof a user within the physical, real-world environment. Consequently, afirst user (e.g., user 106A) may visualize the energy differently than asecond user (e.g., user 106B) because they are positioned at differentlocations within the physical, real-world environment. Therefore, theinput module 116 can also be configured to receive data associated withpositions and orientations of users 106 and their bodies in space (e.g.,tracking data), so the locations of the user within the physical,real-world environment can be determined. Tracking devices can includeoptical tracking devices (e.g., VICON®, OPTITRACK®), magnetic trackingdevices, acoustic tracking devices, gyroscopic tracking devices,mechanical tracking systems, depth cameras (e.g., KINECT®, INTEL®RealSense, etc.), inertial sensors (e.g., INTERSENSE®, XSENS, etc.),combinations of the foregoing, etc. The tracking devices can outputstreams of volumetric data, skeletal data, perspective data, etc. insubstantially real time. The streams of volumetric data, skeletal data,perspective data, etc. can be received by the input module 116 insubstantially real time. Volumetric data can correspond to a volume ofspace occupied by a body of a user. Skeletal data can correspond to dataused to approximate a skeleton, in some examples, corresponding to abody of a user, and track the movement of the skeleton over time.Perspective data can correspond to data collected from two or moreperspectives that can be used to determine an outline of a body of auser from a particular perspective. In some instances, combinations ofthe volumetric data, the skeletal data, and the perspective data can beused to determine body representations corresponding to users 106.

Applications (e.g., application(s) 124) can be created by programmers tofulfill specific tasks. For example, applications (e.g., application(s)124) can provide utility, entertainment, and/or productivityfunctionalities to users 106 of devices 108 or other users. Exampleapplications 124 that can use the techniques described herein includeapplications that use acoustics (e.g., music playing applications, gameapplications with sound effects, etc.), applications associated withenergy efficiency (e.g., visualization of how heat spreads through ahouse), engineering applications (e.g., visualization of how strong aWi-Fi signal is in a house), educational applications (e.g., thatvisualizes how radiation spreads), and so forth.

In some examples, the one or more users 106 can operate correspondingdevices 108 to perform various functions associated with the devices108. Device(s) 108 can represent a diverse variety of device types andare not limited to any particular type of device. Examples of device(s)108 can include but are not limited to stationary computers, mobilecomputers, embedded computers, or combinations thereof. Examplestationary computers can include desktop computers, work stations,personal computers, thin clients, terminals, game consoles, personalvideo recorders (PVRs), set-top boxes, or the like. Example mobilecomputers can include laptop computers, tablet computers, wearablecomputers, implanted computing devices, telecommunication devices,automotive computers, portable gaming devices, media players, cameras,or the like. Example embedded computers can include network enabledtelevisions, integrated components for inclusion in a computing device,appliances, microcontrollers, digital signal processors, or any othersort of processing device, or the like. In at least one example, thedevices 108 can include mixed reality devices (e.g., CANON® MREAL®System, MICROSOFT® HOLOLENS®, etc.). Mixed reality devices can includeone or more sensors and a mixed reality display, as described below inthe context of FIG. 2. In FIG. 1, device 108A and device 108B arewearable computers (e.g., head mount devices); however, device 108Aand/or device 108B can be any other device as described above.Similarly, in FIG. 1, device 108C is a mobile computer (e.g., a tabletdevice); however, device 108C can be any other device as describedabove.

Device(s) 108 can include one or more input/output (I/O) interface(s)coupled to the bus to allow device(s) to communicate with other devicessuch as input peripheral devices (e.g., a keyboard, a mouse, a pen, agame controller, a voice input device, a touch input device, a gesturalinput device, a tracking device, a mapping device, an image camera, adepth sensor, a physiological sensor, a microphone or other acousticsensor, a thermometer, a Geiger counter, a color sensor, a gravitysensor and the like) and/or output peripheral devices (e.g., a display,a printer, audio speakers, a haptic output, and the like).

FIG. 2 is a schematic diagram showing an example of a head mounted mixedreality display device 200. As illustrated in FIG. 2, the head mountedmixed reality display device 200 can include one or more sensors 202 anda display 204. The one or more sensors 202 can include depth camerasand/or sensors, a microphone or other acoustic sensor to capture avoice, inertial sensors, optical sensors, body tracking sensors, etc. Insome examples, as illustrated in FIG. 2, the one or more sensors 202 canbe mounted on the head mounted mixed reality display device 200. The oneor more sensors 202 correspond to inside-out sensing sensors; that is,sensors that capture information from a first person perspective. Inadditional or alternative examples, the one or more sensors can beexternal to the head mounted mixed reality display device 200 and/ordevices 108. Such sensors 202 can also correspond to outside-in sensingsensors; that is, sensors that capture information from a third personperspective.

The display 204 can present visual content (e.g., energy) to the one ormore users 106 in a mixed reality environment. In some examples, thedisplay 204 can present the mixed reality environment to the user (e.g.,user 106A, user 106B, or user 106C) in a spatial region that occupies anarea that is substantially coextensive with a user's (e.g., user 106A,user 106B, or user 106C) actual field of vision. In other examples, thedisplay 204 can present the mixed reality environment to the user (e.g.,user 106A, user 106B, or user 106C) in a spatial region that occupies alesser portion of a user's (e.g., user 106A, user 106B, or user 106C)actual field of vision. The display 204 can include a transparentdisplay that enables a user to view the real scene where he or she isphysically located. Transparent displays can include optical see-throughdisplays where the user sees the real scene he or she is physicallypresent in directly, video see-through displays where the user observesthe real scene in a video image acquired from a mounted camera, etc. Thedisplay 204 can present the virtual content to a user such that thevirtual content augments the physical, real-world scene where the useris physically located within the spatial region.

The devices 108 can include one or more processing unit(s) (e.g.,processor(s) 126), computer-readable media 128, at least including arendering module 130, and one or more applications 132. The one or moreprocessing unit(s) (e.g., processor(s) 126) can represent same unitsand/or perform same functions as processor(s) 112, described above.Computer-readable media 128 can represent computer-readable media 114 asdescribed above. Computer-readable media 128 can include components thatfacilitate interaction between the service provider 102 and the one ormore devices 108. The rendering module 130 can receive rendering data(e.g., visual content such as energy) from the service provider 102 andcan render virtual content on the display 204 of the device. In at leastone example, the rendering module 130 can leverage one or more renderingalgorithms for rendering virtual content on the display 204.Application(s) 132 can correspond to same applications as application(s)124 or different applications.

FIG. 1 further illustrates how propagating energy can be displayed asvirtual content to a user. For instance, user 106C may be viewing,through a display of device 108C, a physical, real-world environmentthat includes a dog sitting on a chair and barking in a living room of ahouse. As the sound of the barking propagates through the room away fromthe source (e.g., the dog), the sound decays. Accordingly, in thisexample, the virtual energy is represented by the presentation of asound wave, and the virtual energy associated with element 134 is louder(e.g., as represented by a portion of the sound wave that is bolder)compared to the virtual energy associated with element 136, whichrepresents sound that is further away from the source. Moreover, thevirtual energy associated with element 136 is louder compared to thevirtual energy associated with element 138, which represents sound thatis even further away from the source. Accordingly, the virtual energyrepresented by elements 134, 136, and 138 illustrates an example of howa user 106C may view invisible energy within a physical, real-worldscene.

An Example Mixed Reality User Interface

FIG. 3 is a schematic diagram 300 showing an example of a first personview of a user (e.g., user 106A) listening to another user (e.g., user106B) talk in a mixed reality environment. The area depicted in thedashed lines corresponds to a physical, real-world scene 302 in which atleast the first user (e.g., user 106A) and the second user (e.g., user106B) are physically present.

In FIG. 3, illustrates sensors 304A, 304B, 304C, which are each placedat a different location in the physical, real-world scene 302. Asdiscussed above, the sensors 304A, 304B, 304C can be configured to scanthe room and report data, to the input module 116 (which can be executedon a device in which one of the sensors 304A, 304B, 304C is integrated),related to the geometry of the room. Using the reported data, thegeometry module 118 can then determine the shape and volumetricinformation associated with objects in the room (e.g., the couch, thechair, the table, the lamp, the bookshelf, etc.) and/or the overallspace or size of the room based on detected boundaries (e.g., the floor,the walls, the ceiling, etc.). Moreover, the geometry module 118 cancompute parameter fields which encode scene level properties, asdiscussed above. The sensors 304A, 304B, 304C can also be configured todetect energy in the room, and report data regarding the detected energyto the input module 116 so that the energy module 120 can combine theenergy data with the geometry data and determine how the energypropagates through the room. The energy can then be rendered as avirtual visualization on a display (e.g., display 204) of a user device(e.g., device 200).

As shown in FIG. 3, user 106B may be talking with user 106A, andtherefore, user 106A can see the sound of user's 106B voice as itpropagates through the room via a display of a corresponding mixedreality device (e.g., device 108A). In this example, the sound isrepresented by spheres. A radius of an individual sphere representsintensity (e.g., loudness) of the sound at that particularthree-dimensional point within the physical, real-world scene.Accordingly, the intensity of the sound decays as it propagates awayfrom the source (e.g., the mouth of user 106B). For instance, a radiusof a first sphere that is further away from the mouth of user 106B issmaller than a radius of a second sphere that is closer to the mouth ofuser 106B.

In some examples, a range of energy frequencies can be divided intobands (e.g., three bands) and an individual band can be assigned a color(e.g., red, green, blue). For instance, the current sound's spectrum canbe converted into RGB values by having a direct relationship between thestrength of frequencies in each band to the intensity of color theycorrespond to. This color information can then be superimposed on a mapgenerated to represent wave propagation. Thus, in addition to seeingdifferent sizes of spheres representing intensity (e.g., loudness), amap can also dynamically change color to indicate the current spectrumof sound.

The techniques described herein can also visualize, as virtual content,multiple different types of energy simultaneously. For example, if thedog 306 is barking while the user 106B is talking, the mixed realityscene can also display energy representing the sound of the dog'sbarking. In some instances, a second energy propagation can berepresented a different shape (e.g., diamonds, squares, or waves—ifspheres are already being used for the first energy propagation).Detecting multiple energy sources can be done using a blind sourceseparation algorithm, for example. In noisy environments (with multiplesound sources), the energy module 120 can identify and select aparticular sound source (e.g., for rendering as virtual content) basedon a known energy signature of the source.

In various examples, the user 106A can see the original sound wave(e.g., represented by spheres) as it originates from a source andexpands until the sound meets an obstacle such as furniture, walls, orfloors. At this point, the sound can be reflected and the user 106A cansee the reflected sound (e.g., a secondary or reflected sound wave). Thesecondary or reflected wave can then propagate until it meets anobstacle. This process of propagation and reflection can be visualizeduntil the intensity of the sound is below a certain intensity thresholdat which point the sound will no longer be displayed.

In various examples, the energy module 120 can be configured toartificially slow down the propagation of energy so that a user can viewthe energy. For instance, a timescale associated with the energypropagation can be distorted to match the human sensory and cognitivesystems.

FIG. 4 is a schematic diagram 400 showing an example of a gridrepresenting the propagation of energy through a three-dimensionalspace. As seen in FIG. 4, the sphere-like representation of energy atthe center of the grid appears stronger than the sphere likerepresentation of the energy at the edges of the grid. Thus, a source ofenergy in the grid is likely at the center of the grid. Consequently,the grid could correspond to the physical, real-world scene 302 of FIG.3, in which the source of sound energy is the mouth of user 106B.

Example Processes

The processes described in FIGS. 5 and 6 below are illustrated as acollection of blocks in a logical flow graph, which represent a sequenceof operations that can be implemented in hardware, software, or acombination thereof. In the context of software, the blocks representcomputer-executable instructions stored on one or more computer-readablestorage media that, when executed by one or more processors, perform therecited operations. Generally, computer-executable instructions includeroutines, programs, objects, components, data structures, and the likethat perform particular functions or implement particular abstract datatypes. The order in which the operations are described is not intendedto be construed as a limitation, and any number of the described blockscan be combined in any order and/or in parallel to implement theprocesses.

FIG. 5 is a flow diagram that illustrates an example process 500 todetermine a representation of how sensed energy propagates from alocation of a source through the physical, real-world environment and tovisualize the representation. The process can be implemented by one ormore of the servers, devices, and/or sensors, as discussed above withrespect to any one of FIGS. 1-3.

At 502, an environment is scanned to determine spatial objects and/orboundaries that affect the propagation of energy within the environment.For example, sensors positioned throughout the environment can beinstructed to scan the environment for geometry data and report the datato the input module 116 (e.g., the input module 116 receives thegeometry data from the sensors).

At 504, geometry-based properties are computed based on the datacaptured via the scan. For example, the geometry module 118 can computeparameter fields which encode scene level properties such as one or moreof a loudness map (e.g., for sound propagation), a reflection map tomodel how energy is redirected in response to encountering a real or avirtual object, a reverberation map to model how repeated energy (e.g.,an echo) propagates, an amount of time it takes energy to travel from asource location (e.g., a first three-dimensional point) in theenvironment to another three-dimensional point within the real scene,etc.

At 506, energy is sensed and/or data associated with the sensed energyis reported. For example, sensors positioned throughout the environmentcan be configured to monitor for energy and report data associated withthe detected energy to the input module 116 (e.g., the input module 116receives the energy data from the sensors). In some instances, the scandiscussed above with respect to 504 can be implemented in response toenergy being detected.

At 508, a source of the energy is located. For example, triangulationcan be used to locate a source of sound or other energy.

At 510, the geometry-based properties are used to compute arepresentation of how the sensed energy propagates from the location ofthe source of the energy through the environment.

For example, a finite-difference wave equation simulator can be used tocompute the representation. If a detected energy source is located atsome position, x₀, the signal at the source at a particular time can berepresented by (x₀, t). A propagation model can provide properties ofthe propagation of energy at a three-dimensional point (e.g., any and/orevery point) in space due to the source, which can be represented asA(x, x₀). If the time-delay for propagation is τ(x, x₀), a geodesic pathdelay around scene geometry can be represented as τ(x, x₀)=g(x, x₀)/c ora line of sight can be represented as τ(x, x₀)=|x−x₀|/c. Then, thefollowing function can be computed: Color(x, t)=R(s(x₀,t−τ(x, x₀)), A(x,x₀)). Here R is a function that combines the temporal, time-delayedsource signal that arrives at the point and the propagationcharacteristics at that point.

In some examples, the energy module 120 can divide a three-dimensionalspace (e.g., the room of FIG. 3) in which energy propagates into anoccupancy grid (e.g., the 3-D space is discretized into an array). Theoccupancy grid can describe a history of vector fields. Each element ofthe occupancy grid can be associated with historical properties thatdescribe the propagation of energy that entered the system at differenttime instants. Thus, for an individual occupancy grid, data associatedwith properties of energy (e.g., intensity, direction, frequencyrepresented by color, etc.) that entered the system at various times(e.g., t, t−1, t−2, and so forth) can be maintained. Accordingly, for acomputation cycle, the following steps of an example algorithm canoccur:

-   -   1. Introduce detected energy: From the detected energy source        location, the surrounding voxels in the occupancy grid can be        assigned seed values, for example, an initial vector        (intensity+direction) and color representing frequency.    -   2. Propagating energy:        -   a. Update step: For each past time instant and for each            voxel in the occupancy grid, a new vector and color can be            calculated as a sum of contributions of the vectors and            colors of the neighboring voxels from a previous time            instant. This can also take into account the latest            parameter fields to modulate the propagating energy. For            example:            -   i. Vector (x, y, z, t)=(vector sum of neighboring vector                field from time t−1)*modulation by latest parameter                field            -   ii. Color (x, y, z, t)=Original Color*step function                (vector amplitude)        -   b. Summing contributions: At each voxel, a sum of the            contributions of the vector field and color for each time            instant can be calculated and this can provide, for example,            a radius of a sphere based on an amount of intensity at a            particular voxel location and a color of the sphere at a            particular voxel location.

At 512, the representation is virtualized and/or rendered for display soa user can view invisible energy, for example, in a mixed reality scene.For instance, rendering data can be sent by server(s) 110 to a mixedreality device 200 associated with a user so the rendering module 130can render the representation to be displayed as virtual content (e.g.,the representation overlays a real-world view of the user).

In various examples, operations 506 through 512 can be repeated ifanother energy from another source is detected in the environment (e.g.,a person talking and a dog barking). In some instances, the detectedenergies in an environment can be of different types (e.g., sound, heat,light, radiation, electro-magnetic field, etc.). Thus, differentenergies can be rendered differently so a user, while viewing, candistinguish between them (e.g., a first energy can be represented by aset of spheres while a second energy can be represented by a set ofsquares).

FIG. 6 is a flow diagram that illustrates an example process 600 thatre-computes the representation of how the sensed energy propagates fromthe location of the source of the energy through the environment basedon a viewing perspective of a user. The process can be implemented byone or more of the servers, devices, and/or sensors, as discussed abovewith respect to any one of FIGS. 1-3.

At 602, a location and an orientation of a device associated with a user(e.g., a mixed reality device) located in the environment is determined.For example, the geometry module 118 can determine that a device of afirst user (e.g., user 106A from FIG. 3) is located directly in front ofa second user (e.g., user 106B in FIG. 3) and that the device isdirected so that it is facing the second user.

At 604, a viewing perspective of the user is determined based on thelocation and the orientation of the device. For instance, the viewingperspective in FIG. 3 is looking at user 106B instead of looking awayfrom user 106B.

At 606, the representation of how the sensed energy propagates from thelocation of the source of the energy through the environment isre-computed based on the viewing perspective of the user. For example,user 106A, being located directly in front of user 106B in FIG. 3 viewsa different representation than another user (e.g., user 106C) that islocated behind user 106B. That is, user 106A may visualize a strongerrepresentation of the voice of user 106B compared to the representationvisualized by user 106C, since user 106B is facing and talking to user106A and not user 106C.

Example Clauses

Example A, a system comprising: one or more processors; memory; and oneor more modules stored in the memory and executable by the one or moreprocessors to perform operations comprising: receiving first dataassociated with real objects and boundaries that affect the propagationof energy within the environment; computing, based at least in part onthe received first data, geometry-based properties; receiving seconddata associated with energy in the environment; locating, based at leastin part on the received second data, a source of the energy within theenvironment; using the geometry-based properties to compute arepresentation of how the energy propagates from a location of thesource of the energy through the environment; and causing therepresentation of how the energy propagates from the location of thesource of the energy through the environment to be rendered on adisplay.

Example B, the system as Example A recites, wherein the display is partof, or coupled to a mixed reality device, and the display comprises atransparent display.

Example C, the system as Example A or Example B recites, wherein therepresentation comprises a set of spheres distributed throughout theenvironment and a radius of an individual sphere is determined based onan intensity of the energy at a particular three-dimensional locationwithin the environment.

Example D, the system as Example C recites, wherein a first sphere inthe set of spheres has a first radius that is greater than a secondradius of a second sphere in the set of spheres, the first sphere beingcloser to the source of the energy than the second sphere.

Example E, the system as any one of Example A through Example D recites,wherein the representation comprises a set of spheres distributedthroughout the environment and a color of an individual sphere isdetermined based on a frequency of the energy at a particularthree-dimensional location within the environment.

Example F, the system as any one of Example A through Example E recites,wherein the geometry-based properties comprise at least one of: aloudness map for sound propagation; a reflection map to model how theenergy is redirected in response to encountering a real or a virtualobject; a reverberation map to model how repeated energy propagates; oran amount of time it takes energy to travel from a source location toanother three-dimensional point within the environment.

Example G, the system as any one of Example A through Example F recites,the operations further comprising: receiving third data associated withanother energy in the environment; locating, based at least in part onthe received third data, another source of the other energy within theenvironment; using the geometry-based properties to compute anotherrepresentation of how the other energy propagates from a location of theother source of the other energy through the environment; and causingthe other representation of how the other energy propagates from thelocation of the other source of the other energy through the environmentto be rendered on the display simultaneously with the representation ofhow the energy propagates from the location of the source of the energythrough the environment.

Example H, the system as any one of Example A through Example G recites,determining a location and an orientation of a mixed reality deviceassociated with a user in the environment; determining a viewingperspective of the user based at least in part on the location and theorientation of the mixed reality device; and using the geometry-basedproperties to re-compute the representation of how the energy propagatesfrom the location of the source of the energy through the environmentbased at least in part on the viewing perspective of the user.

Example I, the system as any one of Example A through Example H recites,wherein the energy is associated with one of: sound; a Wi-Fi signal;heat; light; fluid dynamics; an electro-magnetic field; radiation; or anair current,

While Example A through Example I are described above with respect to asystem, it is understood in the context of this document that thecontent of Example A through Example I may also be implemented via adevice, computer storage media, and/or a method.

Example J, a method comprising: receiving first data associated withreal objects and boundaries that affect the propagation of energy withinthe environment; computing, by one or more processors and based at leastin part on the received first data, geometry-based properties; receivingsecond data associated with energy in the environment; locating, basedat least in part on the received second data, a source of the energywithin the environment; using the geometry-based properties to compute arepresentation of how the energy propagates from a location of thesource of the energy through the environment; and causing therepresentation of how the energy propagates from the location of thesource of the energy through the environment to be rendered on adisplay.

Example K, the method as Example J recites, wherein the display is partof, or coupled to a mixed reality device, and the display comprises atransparent display.

Example L, the method as Example J or Example K recites, wherein therepresentation comprises a set of spheres distributed throughout theenvironment and a radius of an individual sphere is determined based onan intensity of the energy at a particular three-dimensional locationwithin the environment.

Example M, the method as Example L recites, wherein a first sphere inthe set of spheres has a first radius that is greater than a secondradius of a second sphere in the set of spheres, the first sphere beingcloser to the source of the energy than the second sphere.

Example N, the method as any one of Example J through Example M recites,wherein the representation comprises a set of spheres distributedthroughout the environment and a color of an individual sphere isdetermined based on a frequency of the energy at a particularthree-dimensional location within the environment.

Example O, the method as any one of Example J through Example N recites,wherein the geometry-based properties comprise at least one of: aloudness map for sound propagation; a reflection map to model how theenergy is redirected in response to encountering a real or a virtualobject; a reverberation map to model how repeated energy propagates; oran amount of time it takes energy to travel from a source location toanother three-dimensional point within the environment.

Example P, the method as any one of Example J through Example O recites,further comprising: determining a location and an orientation of a mixedreality device associated with a user in the environment; determining aviewing perspective of the user based at least in part on the locationand the orientation of the mixed reality device; and using thegeometry-based properties to re-compute the representation of how theenergy propagates from the location of the source of the energy throughthe environment based at least in part on the viewing perspective of theuser.

Example Q, a system configured to communicate with a mixed realitydevice located within a real-world scene, the system comprising: one ormore sensors; one or more processors; and memory storing instructionsthat, when executed on the one or more processors, cause the system toperform operations comprising: scanning, using the one or more sensors,the real-world scene to obtain first data associated with real objectsthat affect the propagation of energy within the real-world scene;computing, based at least in part on the obtained first data,geometry-based properties of the real-world scene; detecting, using theone or more sensors, second data associated with energy in thereal-world scene; locating, based at least in part on the detectedsecond data, a source of the energy within the real-world scene; usingthe geometry-based properties to compute a representation of how theenergy propagates from a location of the source of the energy throughthe real-world scene; generating rendering data for the representationof how the energy propagates from the location of the source of theenergy through the real-world scene; and sending the rendering data to amixed reality device to be rendered as virtual content in associationwith a view of the real-world scene.

Example R, the system as Example Q recites, wherein the representationcomprises a set of spheres distributed over the real-world scene and aradius of an individual sphere is determined based on an intensity ofthe energy at a particular three-dimensional location within thereal-world scene.

Example S, the system Example R recites, wherein a first sphere in theset of spheres has a first radius that is greater than a second radiusof a second sphere in the set of spheres, the first sphere being closerto the source of the energy than the second sphere.

Example T, the system as any one of Example Q through Example S recites,wherein the representation comprises a set of spheres distributed overthe real-world scene and a color of an individual sphere is determinedbased on a frequency of the energy at a particular three-dimensionallocation within the real-world scene.

While Example Q through Example T are described above with respect to asystem, it is understood in the context of this document that thecontent of Example Q through Example T may also be implemented via amethod, computer storage media, and/or a device.

Example U, a system comprising: means for receiving first dataassociated with real objects and boundaries that affect the propagationof energy within the environment and second data associated with energyin the environment; means for computing, based at least in part on thereceived first data, geometry-based properties; means for locating,based at least in part on the received second data, a source of theenergy within the environment; means for using the geometry-basedproperties to compute a representation of how the energy propagates froma location of the source of the energy through the environment; andmeans for causing the representation of how the energy propagates fromthe location of the source of the energy through the environment to berendered on a display.

Example V, a system configured to communicate with a mixed realitydevice located within a real-world scene, the system comprising: meansfor scanning the real-world scene to obtain first data associated withreal objects that affect the propagation of energy within the real-worldscene; means for computing, based at least in part on the obtained firstdata, geometry-based properties of the real-world scene; means fordetecting second data associated with energy in the real-world scene;means for locating, based at least in part on the detected second data,a source of the energy within the real-world scene; means for using thegeometry-based properties to compute a representation of how the energypropagates from a location of the source of the energy through thereal-world scene; means for generating rendering data for therepresentation of how the energy propagates from the location of thesource of the energy through the real-world scene; and means for sendingthe rendering data to a mixed reality device to be rendered as virtualcontent in association with a view of the real-world scene.

CONCLUSION

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are described as illustrative forms ofimplementing the claims.

Conditional language such as, among others, “can,” “could,” or “might,”unless specifically stated otherwise, are understood within the contextto present that certain examples include, while other examples do notnecessarily include, certain features, elements and/or steps. Thus, suchconditional language is not generally intended to imply that certainfeatures, elements and/or steps are in any way required for one or moreexamples or that one or more examples necessarily include logic fordeciding, with or without input or prompting, whether certain features,elements and/or steps are included or are to be performed in anyparticular example. Conjunctive language such as the phrase “at leastone of X, Y or Z,” unless specifically stated otherwise, is to beunderstood to present that an item, term, etc. can be either X, Y, or Z,or a combination thereof.

What is claimed is:
 1. A system comprising: one or more processors;memory; and one or more modules stored in the memory and executable bythe one or more processors to perform operations comprising: receivingfirst data associated with real objects and boundaries that affect thepropagation of energy within the environment; computing, based at leastin part on the received first data, geometry-based properties; receivingsecond data associated with energy in the environment; locating, basedat least in part on the received second data, a source of the energywithin the environment; using the geometry-based properties to compute arepresentation of how the energy propagates from a location of thesource of the energy through the environment; and causing therepresentation of how the energy propagates from the location of thesource of the energy through the environment to be rendered on adisplay.
 2. The system as claim 1 recites, wherein the display is partof, or coupled to a mixed reality device, and the display comprises atransparent display.
 3. The system as claim 1 recites, wherein therepresentation comprises a set of spheres distributed throughout theenvironment and a radius of an individual sphere is determined based onan intensity of the energy at a particular three-dimensional locationwithin the environment.
 4. The system as claim 3 recites, wherein afirst sphere in the set of spheres has a first radius that is greaterthan a second radius of a second sphere in the set of spheres, the firstsphere being closer to the source of the energy than the second sphere.5. The system as claim 1 recites, wherein the representation comprises aset of spheres distributed throughout the environment and a color of anindividual sphere is determined based on a frequency of the energy at aparticular three-dimensional location within the environment.
 6. Thesystem as claim 1 recites, wherein the geometry-based propertiescomprise at least one of: a loudness map for sound propagation; areflection map to model how the energy is redirected in response toencountering a real or a virtual object; a reverberation map to modelhow repeated energy propagates; or an amount of time it takes energy totravel from a source location to another three-dimensional point withinthe environment.
 7. The system as claim 1 recites, the operationsfurther comprising: receiving third data associated with another energyin the environment; locating, based at least in part on the receivedthird data, another source of the other energy within the environment;using the geometry-based properties to compute another representation ofhow the other energy propagates from a location of the other source ofthe other energy through the environment; and causing the otherrepresentation of how the other energy propagates from the location ofthe other source of the other energy through the environment to berendered on the display simultaneously with the representation of howthe energy propagates from the location of the source of the energythrough the environment.
 8. The system as claim 1 recites, theoperations further comprising: determining a location and an orientationof a mixed reality device associated with a user in the environment;determining a viewing perspective of the user based at least in part onthe location and the orientation of the mixed reality device; and usingthe geometry-based properties to re-compute the representation of howthe energy propagates from the location of the source of the energythrough the environment based at least in part on the viewingperspective of the user.
 9. The system as claim 1 recites, wherein theenergy is associated with one of: sound; a Wi-Fi signal; heat; light;fluid dynamics; an electro-magnetic field; radiation; or an air current.10. A method comprising: receiving first data associated with realobjects and boundaries that affect the propagation of energy within theenvironment; computing, by one or more processors and based at least inpart on the received first data, geometry-based properties; receivingsecond data associated with energy in the environment; locating, basedat least in part on the received second data, a source of the energywithin the environment; using the geometry-based properties to compute arepresentation of how the energy propagates from a location of thesource of the energy through the environment; and causing therepresentation of how the energy propagates from the location of thesource of the energy through the environment to be rendered on adisplay.
 11. The method as claim 10 recites, wherein the display is partof, or coupled to a mixed reality device, and the display comprises atransparent display.
 12. The method as claim 10 recites, wherein therepresentation comprises a set of spheres distributed throughout theenvironment and a radius of an individual sphere is determined based onan intensity of the energy at a particular three-dimensional locationwithin the environment.
 13. The method as claim 12 recites, wherein afirst sphere in the set of spheres has a first radius that is greaterthan a second radius of a second sphere in the set of spheres, the firstsphere being closer to the source of the energy than the second sphere.14. The method as claim 10 recites, wherein the representation comprisesa set of spheres distributed throughout the environment and a color ofan individual sphere is determined based on a frequency of the energy ata particular three-dimensional location within the environment.
 15. Themethod as claim 10 recites, wherein the geometry-based propertiescomprise at least one of: a loudness map for sound propagation; areflection map to model how the energy is redirected in response toencountering a real or a virtual object; a reverberation map to modelhow repeated energy propagates; or an amount of time it takes energy totravel from a source location to another three-dimensional point withinthe environment.
 16. The method as claim 10 recites, further comprising:determining a location and an orientation of a mixed reality deviceassociated with a user in the environment; determining a viewingperspective of the user based at least in part on the location and theorientation of the mixed reality device; and using the geometry-basedproperties to re-compute the representation of how the energy propagatesfrom the location of the source of the energy through the environmentbased at least in part on the viewing perspective of the user.
 17. Asystem configured to communicate with a mixed reality device locatedwithin a real-world scene, the system comprising: one or more sensors;one or more processors; and memory storing instructions that, whenexecuted on the one or more processors, cause the system to performoperations comprising: scanning, using the one or more sensors, thereal-world scene to obtain first data associated with real objects thataffect the propagation of energy within the real-world scene; computing,based at least in part on the obtained first data, geometry-basedproperties of the real-world scene; detecting, using the one or moresensors, second data associated with energy in the real-world scene;locating, based at least in part on the detected second data, a sourceof the energy within the real-world scene; using the geometry-basedproperties to compute a representation of how the energy propagates froma location of the source of the energy through the real-world scene;generating rendering data for the representation of how the energypropagates from the location of the source of the energy through thereal-world scene; and sending the rendering data to a mixed realitydevice to be rendered as virtual content in association with a view ofthe real-world scene.
 18. The system as claim 17 recites, wherein therepresentation comprises a set of spheres distributed over thereal-world scene and a radius of an individual sphere is determinedbased on an intensity of the energy at a particular three-dimensionallocation within the real-world scene.
 19. The system as claim 18recites, wherein a first sphere in the set of spheres has a first radiusthat is greater than a second radius of a second sphere in the set ofspheres, the first sphere being closer to the source of the energy thanthe second sphere.
 20. The system as claim 17 recites, wherein therepresentation comprises a set of spheres distributed over thereal-world scene and a color of an individual sphere is determined basedon a frequency of the energy at a particular three-dimensional locationwithin the real-world scene.